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Abstract 

Objective. Despite widespread interest in many jurisdictions in monitoring and improving the quality of stroke care delivery, 
benchmarks for most stroke performance indicators have not been established. The objective of this study was to develop data- 
derived benchmarks for acute stroke quality indicators. 

Design. Nine key acute stroke quality' indicators were selected from the Canadian Stroke Best Practice Performance Measures 
Manual. 

Participants. A population-based retrospective sample of patients discharged from 142 hospitals in Ontario, Canada, between 1 
April 2008 and 31 March 2009 (N = 3191) was used to calculate hospital rates of performance and benchmarks. 

Intervention. The Achievable Benchmark of Care (ABC™) methodology was used to create benchmarks based on the perform- 
ance of the upper 1 5% of patients in the top-performing hospitals. 

Main Outcome Measures. Benchmarks were calculated for rates of neuroimaging, carotid imaging, stroke unit admission, dys- 
phasia screening and administration of stroke-related medications. 

Results. The following benchmarks were derived: neuroimaging within 24 h, 98%; admission to a stroke unit, 77%; thrombolysis 
among patients arriving within 2.5 h, 59%; carotid imaging, 93%; dysphagia screening, 88%; antithrombotic therapy, 98%; antic- 
oagulation for atrial fibrillation, 94%; antihypertensive therapy, 92% and lipid-lowering therapy, 77%. ABC™ acute stroke care 
benchmarks achieve or exceed the consensus-based targets required by Accreditation Canada, with the exception of dysphagia 
screening. 

Conclusions. Benchmarks for nine hospital-based acute stroke care quality' indicators have been established. These can be used 
in the development of standards for quality' improvement initiatives. 

Keywords: benchmarking, measurement of quality, quality improvement, quality management 



Introduction 

Benchmarking has gained popularity as a health care quality' 
performance measurement tool with benchmarks used to 
compare delivery of care across institutions and jurisdictions 
and to encourage excellent performance by ranking institu- 
tions and highlighting top performers [1]. Establishing 
realistic performance benchmarks that monitor implemen- 
tation of evidence-based best practice has been shown to 
improve performance compared with audit and feedback 
alone [2—6]. 



Benchmarks based on subjective or expert panel consensus 
rather than empirical data may be viewed as invalid. 
Data-derived benchmarks based on the average or median by 
definition are unlikely to drive excellence. The Achievable 
Benchmarks of Care (ABC™) approach is a method to estab- 
lish 'real world' performance benchmarks by examining per- 
formance across all relevant organizations or health care 
providers and then determining the best care achieved by at 
least 10% of the eligible patients across organizations to iden- 
tify 'top' performance levels [7, 8]. This method produces 
benchmarks that can be seen as realistic targets, as they have 
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been achieved by at least one provider caring for at least 10% 
of all eligible patients in the sample. 

Currendy, there is limited information on appropriate 
benchmarks for acute stroke care delivery [9—1 1], In Ontario, a 
regionally based system of stroke care delivery — the Ontario 
Stroke System — was established in 2000 [12]. Within this 
system, regional stroke centers are accountable for leading the 
implementation of stroke care best practices across a geo- 
graphic region, which includes a number of community hospi- 
tals, rehabilitation facilities (inpatient and ambulatory), 
community-based providers, community support agencies, 
health promotion practitioners, long-term care facilities and 
pre-hospital care providers [12]. The regional stroke centers 
are typically large teaching or academic hospitals with neur- 
ology and neurosurgical services, sophisticated diagnostic 
technologies and annual stroke/transient ischemic attack 
(TLA) volumes ranging from 400 to >1100 per year. The 
established and organized approach to stroke care within 
Ontario provides a unique opportunity to develop and use 
stroke benchmarks for quality improvement. 

Methods 

We used clinical data collected at all acute care institutions in 
the province of Ontario, Canada, and among eleven regional 
stroke centers, by the Registry of the Canadian Stroke 
Network (RCSN). The ABC ™ methodology was used to cal- 
culate benchmarks for quality indicators for acute stroke care 
including: (1) thrombolysis among patients with ischemic 
stroke arriving within 2.5 h of symptom onset, without contra- 
indications, (2) care on an acute stroke unit, (3) neuroimaging 
within 24 h of hospital arrival, (4) carotid imaging among is- 
chemic stroke patients without atrial fibrillation, (5) dysphagia 
screening within 72 h of hospital arrival, (6) antithrombotic 
therapy, (7) anticoagulation for atrial fibrillation, (8) antihyper- 
tensive therapy and (9) lipid-lowering therapy [2] . 

Data source 

The RCSN includes: (1) a periodic random sample audit 
(Ontario Stroke Audit, OSA) of all acute care institutions with 
at least 10 stroke cases annually and (2) data on consecutive 
stroke/TIA patients seen at eleven regional stroke centers in the 
province of Ontario, Canada (wwwrcsn.org). The RCSN is a 
'prescribed' registry under provincial privacy legislation, and 
charts are audited without patient consent. The overall project is 
approved by the Research Ethics Board of Sunnybrook Health 
Sciences Centre as well as the Research Ethics Board of each 
participating stroke center. Data are collected on all aspects of 
acute stroke management, including demographics, comorbid- 
ities, use of the emergency medical services, emergency depart- 
ment and in-hospital processes of care and complications by 
centrally trained neurology research nurses. Chart validation by 
duplicate chart abstraction has shown excellent agreement 
(kappa scores or intra-class correlation coefficients of >0.8) for 
key variables in the database including age, sex, thrombolysis 
administration, stroke unit care and other processes of care [13]. 



Data sample 

Patients of eighteen years of age or older with stroke or TIA 
seen in a hospital emergency department or admitted to hospital 
were identified from administrative databases the National 
Ambulatory Care Reporting System and the Discharge Abstract 
Database maintained by the Canadian Institute for Health 
Information using International Classification of Diseases, 
Tenth Revision (TCD-10-CA). Those assigned codes 160, 161, 
163, 164, H34.1 and G45 (excluding G45.4) were included. To 
calculate the benchmarks, we used data from the 2008/2009 
RCSN Ontario Stroke Audit, which captured patients dis- 
charged from 142 acute care hospitals between 1 April 2008 
and 31 March 2009. Ninety-nine percent of eligible hospitals 
participated in this audit, and a simple random sample of 17% 
of eligible cases was included (« = 3931) with over-sampling at 
low-volume institutions to ensure each institution contributed a 
minimum of 10 cases and 50 cases at smaller specialized stroke 
centers [14]. Because of concern about a small and unequal 
sample size for the development of some benchmarks, second- 
ary analyses used data from consecutive patients seen at 11 re- 
gional stroke centers between 1 April 2006 and 31 March 2008 
(»=8109). 

Acute Stroke Quality of Care Indicators 

Table 1 lists the nine stroke quality indicators evaluated in this 
study. The indicators of stroke care performance are based on 
the Canadian Stroke Strategy's 2008 Performance Measurement 
Manual, are used for reporting within the Ontario Stroke 
System, reflect a subset of indicators identified by Accreditation 
Canada for hospitals seeking stroke care distinction status and 
are also recommended or reported by other organizations in 
other jurisdictions [2, 10, 11, 15—19]. 

Statistical analysis 

We calculated overall indicator performance as the proportion 
of eligible patients that received each stroke care indicator 
rounded to the nearest whole number. We then used the 
ABC™ methodology to calculate benchmarks for each of the 
nine quality indicators. The overall indicator performance was 
assessed using Kiefe et al.'s algorithm to determine the 
minimum sufficient denominator (MSD) (i.e. eligible patients) 
and whether a Bayesian adjustment was needed to adjust for a 
small number of eligible patients [7]. If a hospital's eligible 
patient sample did not meet the MSD (i.e. number of eligible 
patients) for each indicator, we applied a Bayesian adjustment to 
calculate the hospital's adjusted performance fraction described 
by Kiefe et al. [7, 8]. Starting with the highest performing hos- 
pital for the particular indicator and continuing through the 
next highest performing hospital, we cumulatively added each 
hospital's eligible patients until the total number of patients 
represented in the denominator included at least 15% of the 
total eligible patients across all hospitals. We based the bench- 
mark on at least 15% of the total eligible patients across hospi- 
tals because the calculation of the ABC™ benchmark is based 
on crude data and we wanted to increase the number of eligible 
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Table I Acute Stroke Quality of Care Indicators 



Performance indicators 



1. Proportion of suspected stroke/TIA patients who receive a brain CT/MRT within 24 h of hospital arrival to the emergency 
department. 

2. Proportion of stroke/TIA patients treated on a stroke unit at any time during their inpatient stay. 

3. Proportion of ischemic stroke patients who arrive within 2.5 h of symptom onset and receive acute thrombolytic therapy 
(tPA) b (excluding patients with known contraindications). 

4. Proportion of ischemic stroke patients without atrial fibrillation who receive carotid imaging prior to inpatient hospital 
discharge. 

5. Proportion of stroke (excluding TIA, unconscious patients) in patients with documentation that an initial dysphagia screening 
was performed within 72 h of hospital arrival. 

6. Proportion of ischemic stroke/TIA patients who were prescribed antithrombotic c therapy at discharge. 

7. Proportion of ischemic stroke/TIA patients with atrial fibrillation prescribed anticoagulant therapy 1 ' on discharge from acute 
care (excluding patients with contraindications). 

8. Proportion of ischemic stroke/TIA patients who were prescribed antihypertensive therapy at discharge. 

9. Proportion of ischemic stroke/TIA patients who were prescribed lipid-lowering therapy at discharge. 

All indicators exclude unable to determine final diagnosis except indicator 1 . All indicators exclude stroke type unknown except indicator 1 , 2 
and 5. a CT and MRI based on records with admission date/time, scan date/time recorded. b tPA, tissue plasminogen activator. At the time 
the data were collected, the tPA window of time was 3 h. Midway through the time of data collection (September 2008), trial results 
demonstrated the safe therapeutic window for tPA delivery from stroke symptom onset had increased to 4.5 h. However, we chose to base 
it on the longstanding therapeutic window of 3 h as practice change was not expected to change immediately. Antithrombotic therapy 
includes acetylsalicylic acid (ASA), combination ASA and dipyridamole, clopidogrel and warfarin. 



patients and hospitals to include in the benchmark and to 
reduce the influence of hospitals with small numbers of eligible 
patients on the benchmark. The benchmark was determined by 
dividing the total number of patients receiving the best practice 
care by the total number of patients eligible to have received 
best practice care in this subset. See Supplementary material, 
Appendix A for a sample calculation. For each indicator, hos- 
pitals) were considered 'top' performers if their performance 
rate was at or above the ABC™ benchmark. 

The ABC™ process was repeated using the validation cohort 
from 1 1 regional stroke centers, except that there was no need 
to apply the Bayesian adjustment as the number of eligible 
patients exceeded the MSD denominator at all of these hospi- 
tals. For each indicator benchmark, we report the total number 
of hospitals included in our benchmark calculations, as well as 
the range in the number of eligible patients at hospitals included 
in the ABC™ benchmark calculation. For each indicator, we 
report (1) the median, 25th and 75th percentile of eligible 
patients among the hospitals, (2) categories of the number of eli- 
gible patients, <10, 10-24, 25—75, 75 or more and (3) the per- 
centage of hospitals in the audit with eligible patient samples 
below the MSD. 

Results 

Table 2 describes the characteristics of participating hospitals 
and patients in the 2008/09 Ontario stroke audit. Of the 142 
hospitals included in the audit sample, 28 (20%) were regional 
or district stroke centers, 70 (49%) were high volume sites 
(with more than 100 patients with stroke or TIA annually), 90 
(63%) had computed tomography (CT) scanners, 26 (18%) 



had stroke units, 70 (49%) had interdisciplinary stroke teams 
and 39 (28%) had a secondary stroke prevention clinic on site. 
The study sample included 3931 patients with a median age of 
75 years. The number of patients sampled across the 142 hos- 
pitals ranged from 10 tol66. Baseline characteristics of study 
patients are summarized in Table 2. 

There was a wide range in performance for all nine indica- 
tors across acute care hospitals within the OSS. The overall 
performance ranged from 30% (admission to a stroke unit) to 
94% (prescribing of antithrombotic therapy at discharge). 
Only two of the nine indicators (neuroimaging within 24 h and 
antithrombotic therapy prescribed on discharge) had overall 
performance rates of >85% (see Table 3). 

Five of Ontario's nine ABC™ benchmarks for each acute 
stroke quality indicator were >90%: neuroimaging within 24 h 
(either CT or magnetic resonance imaging (MRI) of the brain) 
and antithrombotic therapy prescribed at discharge, 98%; war- 
farin prescribed at discharge for patients with atrial fibrillation, 
94%; carotid imaging prior to discharge, 93% and antihyper- 
tensive therapy prescribed at discharge, 92%. ABC™ bench- 
marks of <90% were dysphagia screening, 88%; admission to 
a stroke unit and lipid-lowering therapy prescribed at discharge 
77% and thrombolysis administration, in patients presenting 
within 2.5 h of stroke symptom onset and without contraindi- 
cations for thrombolysis, 59%. 

The five Accreditation Canada indicator non-data-derived 
targets correspond to two of the ABC™ benchmarks (neuroi- 
maging and antithrombotic prescribing) but are lower for 
stroke unit admission and warfarin prescribing and higher for 
dysphagia screening. Benchmarks derived from regional stroke 
center data were similar to those derived from the entire pro- 
vincial hospital sample for neuroimaging and antithrombotic 
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Table 2 Characteristics of stroke/TIA patients (JV=3931) 
and hospitals (N= 142) in the 2008/09 Ontario Acute Stroke 
Audit 



Characteristic 


N (%) 


Hospital characteristics 


(N = 142) 


Designation: regional stroke centers" 


9(6) 


District stroke centers 1 ' 


19 (13) 


Non-designated hospitals c 


114(80) 


Urban hospitals 


83(58) 


Annual stroke/TIA volume >100 


70 (49) 


Hospitals with stroke units 


26 (18) 


Hospitals with designated stroke teams 


70 (49) 


Hospitals with CT on site 


90 (63) 


Hospitals with secondary stroke prevention 


39 (27) 


clinics on site 




Patient characteristics 


(N = 3 931) 


Stroke 


2 425(62) 


Transient ischemic attack 


1 167(30) 


Unable to determine 


338 (9) 


Stroke type 


(N = 2 370) 


Ischemic 


1896 (80) 


Intracerebral hemorrhage 


308 (13) 


Subarachnoid hemorrhage 


119 (5) 


Undetermined 


24 (1) 


Male 


1965 (50) 


Median age (years, 25th and 75th percentile) 


75 (60, 81) 


Rural residence 


590 (15) 


scores 




Transported by ambulance 


2 240 (57) 


Independent prior to admission 


3 302 (84) 


Diabetes 


983 (25) 


Hypertension 


2 555 (65) 


Hyperlipidemia 


1 454 (37) 


Atrial fibrillation 11 


589 (15) 


Previous TIA/ stroke 


1 336 (34) 


Previous MI 


511 (13) 



"Regional stroke center: all the requirements of a district stroke center, 
plus neurosurgical facilities and interventional radiology. ''District 
stroke center: facilities with written stroke protocols (e.g. transport 
and triage, thrombolytic therapy and neuroimaging), clinicians with 
stroke expertise and linkages to rehabilitation and secondary 
prevention. c Non-designated: acute care hospital that does not fit 
the definition of district or regional stroke center. d Atrial 
fibrillation was based documented on past history of OR new 
onset during hospital stay. 

therapy, higher for stroke unit admission and dysphagia 
screening and lower for all other indicators (Table 3). Less 
than a quarter of all hospitals in the audit (22%) were included 
in the benchmark calculation. The median number of eligible 
patients included in the benchmark calculation ranged from 5 
for warfarin prescribing among patients with atrial fibrillation 
to 51 patients for neuroimaging within 24 h. The number of 
hospitals that met or exceeded the benchmark for any indica- 
tor ranged from 2 hospitals for patients presenting within 2.5 
h of stroke symptom onset and without contraindications for 



thrombolysis to 61 hospitals for prescribing antithrombotic 
therapy on discharge (data not shown). 

Table 4 illustrates the distribution of hospitals by categories 
of eligible patient sample size for each acute stroke quality of 
care performance indicator. The number of hospitals contrib- 
uting to a performance indicator ranged from 93 (for thromb- 
olysis) to 141 (for use of medications for secondary stroke 
prevention). Among our nine acute stroke quality indicators, 
the MSD varied from 4 to 36 eligible patients based on overall 
indicator performance. The percentage of hospitals with eli- 
gible patients below the MSD varied from 14% for stroke unit 
admission to 91% for warfarin prescribing on discharge 
among patients with atrial fibrillation. For each indicator, at 
least 65% of hospitals had <25 eligible patients sampled, and 
for thrombolysis administration, only 1 hospital had at least 25 
eligible patients and for warfarin prescription, no hospitals 
were sampled with at least 25 eligible patients. With the excep- 
tion of prescribing antihypertensive therapy on discharge, all 
of the benchmarks had at least two specialized stroke hospitals 
(regional or district stroke centers) that achieved or exceeded 
the benchmark. 

Discussion 

This study provides benchmarks for nine important acute 
stroke care indicators using the ABC™ methodology with ad- 
justment made for hospitals with small numbers of eligible 
patients, detailed indicator definitions to allow our work to be 
replicated, and the median and range of eligible patients within 
the hospitals included in the benchmark calculation is pro- 
vided to assess the precision of the benchmark. These bench- 
marks are derived from a representative sample of stroke/TIA 
patients seen at 142 hospitals providing stroke care in a prov- 
ince of >13 million people within an organized system of 
stroke care. We also provide the distribution of eligible patients 
for each indicator across the hospitals to assess overall indica- 
tor performance witiiin the Ontario Stroke System. In add- 
ition, we also provide benchmarks based on data from regional 
stroke centers. 

Most of our benchmarks are similar to those found by 
Hinchey et al. [9] using the ABC™ method to calculate stroke 
care benchmarks for ischemic stroke patients from 17 volunteer 
hospitals (13 were community hospitals) across 9 US states (IV 
= 2294). Our neuroimaging, stroke unit admission, prescribing 
antithrombotics and anticoagulants for patients with atrial fibril- 
lation benchmarks are similar to the overall performance levels 
reported in other countries from a subset of hospitals that par- 
ticipate in quality improvement initiatives [11, 15—19]. 
Compared with the consensus-based non-data-derived targets 
used by Accreditation Canada to designate centers of stroke dis- 
tinction, our data-derived ABC ™ benchmarks based on a 
sample of eligible patients across all acute hospitals in Ontario 
meet or exceed the Accreditation Canada targets with the excep- 
tion of dysphagia screening [10]. 

Deciding what benchmark is appropriate depends on the 
purpose of that benchmark. Some might argue we only need to 
have consensus-based targets or performance better than the 
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Table 3. Achievable Benchmarks of Care in Ontario's acute care hospitals (2008/09) 



Stroke process of care Number of Overall Benchmarks 



indicators 


eligible patients 


performance (%) 


OSA- 
ABC™ (%) 


Number of hospitals included in OSA benchmark 
calculation (median, range of eligible patients 
included in the benchmark calculation) 


Accreditation 
Canada targets 
(10 ref) (%) 


Stroke Centre 
ABC™ (%) 


Neuroimaging <24 h 


3176 


86 


98 


11 (51,21-138) 


>90% 


99 


Admitted to a stroke unit 


2457 


30 


77 


10 (46, 4-101) 


75% 


94 


Arrived within 2.5 h of stroke 


469 


30 


59 a 


5 (11,7-33) 


NA 


42 


onset and received tPA a 














Carotid imaging 


1186 


75 


93 


11 (18, 12-56) 


NA 


87 


Dysphagia screening 


1924 


62 


88 


18 (24, 3-66) 


>90% 


90 


Antithrombotics on discharge 


2883 


94 


98 


11 (43,33-100) 


>90% 


97 


Warfarin among patients with 


456 


70 


94 


13 (5,3-15) 


75% 


88 


atrial fibrillation on discharge 














Antihypertensives on discharge 


2883 


78 


92 


31 (16, 6-60) 


NA 


79 


Lipid-lowering agents on 


2883 


60 


77 


22 (27, 3-65) 


NA 


68 



discharge 



"only hospitals with capacity to deliver tPA. OSA, Ontario Stroke Audit; NA, not available. 



Table 4. Distribution of Eligible Patient Volume in the 2008/ 09 Ontario Stroke Audit 



Eligible patients Neuroimaging Stroke unit tPA among patients Carotid Swallowing Discharge medications 

(hospital, N) within 24 h admission arriving within 2.5 h imaging assessment T""i V "•' ^ "'"7 V'"', ■ 7" ", ", •"" 

v r ' ° 00 Antithrombotics Warfarin tor Antihypertensives Lipid-lowenng 

or symptom onset ... 
r patients with 

AF 



0 


2 


4 


49 


10 


7 


1 


36 


1 


1 


<10 


62 


72 


76 


81 


76 


68 


94 


68 


68 


10-24 


34 


26 


16 


43 


35 


25 


12 


25 


25 


25-75 


37 


35 


1 


8 


22 


46 


0 


46 


46 


>75 


7 


5 


0 


0 


2 


2 


0 


2 


2 


Overall 3 


140 


138 


93 


132 


135 


141 


106 


141 


141 


Median eligible 


11 (7, 32) 


9 (5, 27) 


3 (1,7) 


5 (2, 13) 


6 (3, 21) 


10 (7, 34) 


3 (1, 6) 


10 (7, 34) 


10 (7, 34) 


patients (25th and 




















75th percentile) 1 ' 




















MSD 


23 


4 


4 


13 


8 


36 


11 


13 


8 


% Hospitals below 


33 


14 


58 


74 


53 


78 


91 


53 


33 


MSD 




















Number of RSC C at 


2/9 


1/9 


0/9 


0/9 


2/9 


0/9 


0/9 


0/9 


2/9 


or above the 




















ABC™ benchmark 




















Number of DSC d at 


0/19 


3/19 


2/19 


3/19 


1/19 


5/19 


3/19 


0/19 


4/19 



or above the 
ABC™ benchmark 



"Number of hospitals with at least 1 eligible patient. ''Median number of eligible patients among hospitals with at least 1 eligible patient. Regional stroke center: all the requirements of a district 
stroke center, plus neurosurgical facilities and interventional radiology. d District stroke center: facilities with written stroke protocols (e.g. transport and triage, thrombolytic therapy and 
neuroimaging), clinicians with stroke expertise; and linkages to rehabilitation and secondary prevention. MSD, minimum sufficient denominator (i.e. eligible patients). 
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overall average especially if funding is tied to achieving a bench- 
mark. However, under a quality improvement framework where 
data are essential for measuring performance, providing the 
number of eligible patients contributing to the benchmark pro- 
vides organizations full disclosure in assessing the credibility, ac- 
ceptability and achievability of the benchmark [20]. In Ontario, 
the OSS evaluation program has developed report cards on the 
quality of stroke care to be used by the OSS regions, hospitals, 
system planners and provincial agencies to understand the 
strengths and weaknesses and prioritize quality improvement 
initiatives in acute stroke care across Ontario hospitals [12, 21]. 
The benchmarks included in the report card are based on the 
provincial audit data (OSA). Benchmarks will need to be recali- 
brated over time, and the advantage of ABC™ methodology 7 is 
that as all organizations improve the benchmarks rise as well 
while remaining achievable. 

The ABC™ approach to deriving benchmarks is data-driven, 
represents a level of excellence and is demonstrably attainable 
[7]. However, chance fluctuations, small sample sizes or samples 
of unequal size can produce unstable performance measures 
[7, 22—24]. Small and unequal sample size is a limitation of our 
study as it is for most performance measurement studies [22— 
24]. We evaluated the effect of small sample size on our bench- 
mark calculations three ways. Firstly, we excluded facilities with 
<25 eligible patients. This analysis resulted in no change to the 
neuroimaging benchmark, and minor (1 to 4%) decreases to 
two benchmarks and increases to four benchmarks (data not 
shown). In our study, four out of nine benchmarks (neuroima- 
ging, stroke unit admission, antithrombotic and lipid-lowering 
agents prescribed on discharge) were not dominated by hospitals 
with numbers of eligible patients; median eligible patients 
among hospitals contributing to the benchmark were 51, 46, 43 
and 27, respectively (Table 3). Five of our benchmarks were 
driven by small sample sizes therefore may be viewed with 
caution (median number of eligible patients were <25); these 
include warfarin prescribing on discharge among patients with 
atrial fibrillation, thrombolysis, carotid imaging, dysphagia 
screening and antihypertensive medication prescribing these 
benchmarks may be viewed with caution (Table 3). In particular, 
the warfarin prescribing benchmark did not have any hospital 
included in the benchmark calculation with >1 5 eligible patients. 
However, we have presented the median and range of eligible 
patients among the hospitals included in the benchmark calcula- 
tion for the reader to assess the acceptability the benchmark 
based on a sample of all hospitals in Ontario. 

Secondly, we explored whether the benchmarks would 
change if we excluded hospitals where the MSD was not met 
for each indicator. We found all benchmarks remained un- 
changed, with the exception of warfarin prescribing for atrial 
fibrillation where the benchmark was reduced from 94 to 76% 
(data not shown). 

Finally, we repeated the benchmark calculations using data 
from 11 regional stroke centers where eligible patient sample 
sizes are larger and demonstrate less variation. We found that 
six of the nine ABC™ benchmarks were lower (1 to 17%) than 
those derived from the entire sample of hospitals (Table 3). 
However, neuroimaging, admission to a stroke unit and dyspha- 
gia screening benchmarks were higher (1, 17 and 2%, 



respectively). This finding may be a statistical artifact; difference 
in years of data may reflect real differences in processes of care 
at these regional stroke centers. For example, regional stroke 
centers are large tertiary centers and may have facility/ system 
issues that prevent them from achieving higher levels of per- 
formance. Stroke patients compete with trauma and cancer 
patients for access to assessment and diagnostic imaging not 
faced by the smaller district stroke centers. Given the structural 
issues and more complex patients at regional stroke centers, the 
benchmarks derived from regional stroke centers may be con- 
sidered to have better face validity among those centers with 
similar characteristics (e.g. large academic/teaching hospitals) to 
be used for peer benchmarking. Additionally, the time- 
dependent difference between the regional stroke center data 
(2006 to 2008), and the provincial hospital population-based 
data (2008/09) may also explain the lower benchmarks obtained 
using the regional stroke center data. However, when we recali- 
brated the regional stroke center-derived ABC™ benchmarks 
using the same fiscal year data as the OSA ABC™ benchmarks, 
1 April 2008 to 31 March 2009, we did find a significant change 
to four of ABC™ benchmarks based on 2006-2008 data. In 
particular, thrombolysis increased from 42 to 48%, warfarin 
prescribing decreased from 88 to 76% and antihypertensive and 
lipid-lowering prescribing increased from 79 to 86% and 68 to 
82%, respectively. 

Hierarchical modeling has been cited as an appropriate stat- 
istical method to use for performance measurement and in 
particular for outcomes of care rather than processes of care 
to deal with the issue of unstable performance metrics based 
on small and unequal sample sizes [23—25]. Hierarchical mod- 
eling takes the performance estimate of a facility and 'shrinks' 
it closer to the mean performance among all facilities and, the 
degree of 'shrinkage' is greater for facilities with smaller 
sample sizes. A benchmark could be based on the 90th per- 
centile among these 'shrunken' performance estimates derived 
from hierarchical modeling. O'Brien et al, [22] generated 
benchmarks based on the 90th percentile using hierarchical 
modeling and found this 'shrinkage' resulted in minimally 
lower benchmarks (<3%) compared with the benchmarks 
derived using the ABC™ methodology with the Bayesian esti- 
mator to reduce the influence of a small sample for five out of 
their eight ABC™ benchmarks, but more marked (7 to 32% 
lowering) for the remaining three benchmarks with the most 
dramatically lower benchmark for the indicator dominated by 
hospitals with <25 eligible patients (>90% of the hospitals). 
We chose not to use hierarchical modeling to calculate our first 
release of our process of care benchmarks as we wanted to 
provide an easily understood method to stakeholders. The 
ABC™ methodology uses the actual performance with a 
Bayesian adjustment made for facilities that are below the 
MSD to reduce the influence of small sample sizes and has 
greater transparency compared with a statistical modeling ap- 
proach to adjust hospital performance rates. Furthermore, 
O'Brien et a/.'s work demonstrated minimal difference for the 
majority of their process of AMI care benchmarks generated 
by hierarchical modeling compared with the ABC™ bench- 
marks, and Arling et aJ's ischemic stroke quality performance 
indicators demonstrated modest increases in the 90th 
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percentile performance level when applying multilevel model 
empirical Bayes estimation methods compared with the un- 
adjusted 90th percentile performance [22, 24]. Arling et al. 
also observed small denominators (i.e. eligible patients) for 
many of their quality indicators as did our study and consider- 
able uncertainty remained with the estimated Bayes adjusted 
performance indicators. We, however, did not produce confi- 
dence intervals around our ABC™ benchmark because the 
purpose of our benchmarks is to drive quality improvement 
rather than identify good versus poor performance. Finally, 
our quality indicators and benchmarks reflect the Canadian 
context, and therefore our benchmarks may not be transfer- 
able to other jurisdictions. The data were collected to measure 
performance based on the Canadian perspective according to 
the Canadian Stroke Strategy Performance Measurement 
Handbook, which was designed to measure the implementa- 
tion of the Canadian Stroke Strategy Best Practice 
Recommendations (2). 

Conclusion 

Benchmarks are an important tool for quality improvement 
initiatives and empirically derived benchmarks including 
patient sample size offers transparency to allow the credibility 
of the benchmarks to be assessed. This is the first Canadian 
study to report benchmarks from all acute care hospitals 
within a large province. Although the acute hospitals vary 
widely in size and complexity of service provision, these 
benchmarks for acute stroke care delivery can be used for 
reporting and quality improvement to strive for excellence in 
delivering acute stroke care. Further research is needed to 
examine what is the optimal benchmark for treatments that 
have high prevalence of contraindications such as anticoagula- 
tion in atrial fibrillation and thrombolysis. 
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